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IMAGE ENHANCEMENT 

Government License Rights 

The U.S. Government has a paid-up license in this invention and the right 
in limited circumstances to require the patent owner to license others on 
reasonable terms as provided for by the terms of Contract No. GSA D.0.F30602- 
5 00-FA001 awarded by the Air Force Research Laboratory (AFRL). 

Related Application 

The present application is related to co-pending patent application entitled 
"VIDEO MOSAIC", (Docket No. 3351-042) and assigned to the instant assignee 
and filed on even date herewith and is hereby incorporated by reference in its 
1 0 entirety into this specification in its entirety. 

Field of the Invention 

The present invention is related generally to video mosaics, and more 
particularly, to an image enhancement process used in conjunction with a video 
mosaic process. 

15 Background of the Invention 

Until recently, image processing systems have generally processed 
images, such as frames of video, still photographs, and the like in an individual 
manner. After processing, the individual images are combined to form a mosaic, 
i.e., an image that contains a plurality of individual images. Additional image 
20 processing is performed on the mosaic to ensure that the seams between the 
images are invisible such that the mosaic looks like a single large image. The 
alignment was previously done manually by a technician to remove the seams. In 
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such computer aided processing systems, the technician manually selects 
processed images, manually aligns those images, and a computer applies various 
images combining processes to the images to remove any seams or gaps between 
the images. 

5 One problem that has been noted with respect to video mosaics is that the 

video mosaics produce low resolution images because the video images are low 
resolution images. One proposed solution is disclosed in a final Technical Report 
No. RL-TR-96-21, entitled "Multiframe Integration for High-Resolution Video 
Stills", by Dr. Robert L. Stevenson and Dr. Richard R. Schultz. Although the 
10 solution proposed in this report provides excellent results, the drawback to the 
solution is there requires significant processing complexity. A need still exists in 
the art for an image enhancement process which can be performed by computer 
processing equipment mounted on an unmanned ariel vehicle (UAV). 

15 Summary of the Invention 

It is, therefore, an object of the present invention to provide an image 
enhancement process and system which requires minimal computer processing 
capability. 

It is another object of the present invention to provide an image 
20 enhancement process and system which can be performed on-board a vehicle such 
as a UAV taking a sequence of video images. 

Another object of the present invention is to provide an image 
enhancement process in which the sequence of video images is upsampled. 
Registration is then performed to align the images and then one enhanced video 
25 image is formed. 

These and other objects of the present invention are achieved by a 
computer-implemented method of enhancing a video image. A sequence of video 
frames is extracted. Each of the video frames is upsampled. The upsampled 
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video frames are interpolated. The interpolated video frames are aligned and a 
single image is created from the aligned video frames. 

The foregoing and other objects of the present invention are achieved by a 
computer architecture. A sequence of video frames is extracted, each of the 
5 video frames is upsampled. The upsampled video frames are interpolated. The 
interpolated video frames are aligned and a single image is created from the 
aligned frames. 

The foregoing and other objects of the present invention are achieved by 
an article including at least one sequence of machine executable instructions. A 

10 medium bears the executable instructions in machine form and executes the 
instructions by one or more processors caused by the one or more processors. A 
sequence of video frames is extracted. Each of the video frames is upsampled. 
The upsampled video frames are interpolated. The interpolated video frames are 
aligned and a single image is created from the aligned video frames. 

15 The foregoing and other objects of the present invention are achieved by a 

computer system, a processor and a memory coupled to the processor. The 
memory has stored sequences of instructions which when executed by the 
processor causes the processor to perform the following. A sequence of video 
frames is extracted. Each of the video frames is upsampled. The upsampled 

20 video frames are interpolated. The interpolated video frames are aligned and a 
single image is created from the aligned video frames. 

Still other objects and advantages of the present invention will become 
readily apparent to those skilled in the art from the following detailed description, 
wherein the preferred embodiments of the invention are shown and described, 

25 simply by way of illustration of the best mode contemplated of carrying out the 
invention. As will be realized, the invention is capable of other and different 
embodiments, and its several details are capable of modifications in various 
obvious respects, all without departing from the invention. Accordingly, the 
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drawings and description thereof are to be regarded as illustrative in nature, and 
not as restrictive. 



Brief Description of the Drawings 

5 The present invention is illustrated by way of example, and not by 

limitation, in the figures of the accompanying drawings, wherein elements having 
the same reference numeral designations represent like elements throughout and 
wherein: 

Figure 1 is a high-level block diagram of a computer architecture usable 
1 0 with the present invention; 

Figures 2A and 2B are illustrations of a sequence of two consecutive video 

frames; 

Figures 3A and 3B are illustrations of a sequence of two consecutive edge 
detected frames corresponding to Figures 2 A and 2B; 
15 Figures 4 A and 4B are illustrations of a sequence of consecutive frames 

with structures identified corresponding to Figures 3 A and 3B; 

Figures 5A and 5B are illustrations of consecutive frames showing 
matched structures; 

Figures 6A and 6B are flow diagrams according to the present invention of 
20 constructing a video mosaic; 

Figure 7 is a flow diagram of matching identified structures; 

Figure 8 is a flow diagram of edge detection; 

Figure 9 is a series of five resolution images; 

Figure 10 is an upsampled sparsely populated matrix corresponding to the 
25 images of Figure 7; 

Figure 11 is an enlargement of adjacent pixels in a small section from 
Figure 10; 

Figure 12 are five interpolated images corresponding to Figure 10; 
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Figure 13 is a final image after x 5 y alignment and filtering; and 
Figure 14 is a flow diagram according to the present invention of 
enhancing a video mosaic. 



5 Best Mode for Carrying Out the Invention 

A method and apparatus for enhancing an image from a video mosaic 
taken are described. In the following description, for purposes of explanation, 
numerous specific details are set forth in order to provide a thorough 
understanding of the present invention. It will be apparent, however, that the 
10 present invention may be practiced without these specific details. In other 
instances, well-known structures and devices are shown in block diagram form in 
order to avoid unnecessarily obscuring the present invention. 

Hardware Overview 

15 Figure 1 is a block diagram illustrating an exemplary computer system 

100 upon which an embodiment of the invention may be implemented. The 
present invention is usable with currently available personal computers, mini- 
mainframes and the like. Advantageously, the present invention reduces the 
amount of processing capability required and can be processed on-board a UAV. 

20 Computer system 100 includes a bus 102 or other communication 

mechanism for communicating information, and a processor 104 coupled with the 
bus 102 for processing information. Computer system 100 also includes a main 
memory 106, such as a random access memory (RAM) or other dynamic storage 
device, coupled to the bus 102 for storing information and instructions to be 

25 executed by processor 104. Main memory 106 also may be used for storing 
temporary variables or other intermediate information during execution of 
instructions to be executed by processor 104. Computer system 100 further 
includes a read only memory (ROM) 108 or other static storage device coupled to 
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the bus 102 for storing static information and instructions for the processor 104. 
A storage device 110, such as a magnetic disk or optical disk, is provided and 
coupled to the bus 102 for storing information and instructions. 

Computer system 100 may be coupled via the bus 102 to a display 112, 
5 such as a cathode ray tube (CRT) or a flat panel display, for displaying 
information to a computer user. An input device 114, including alphanumeric and 
other keys, is coupled to the bus 102 for communicating information and 
command selections to the processor 104. Another type of user input device is 
cursor control 116, such as a mouse, a trackball, or cursor direction keys for 

10 communicating direction information and command selections to processor 104 
and for controlling cursor movement on the display 112. This input device 
typically has two degrees of freedom in two axes, a first axis (e.g., x) and a 
second axis (e.g.,) allowing the device to specify positions in a plane. 

The invention is related to the use of a computer system 100, such as the 

15 illustrated system, to display an enhanced image of a video mosaic. According to 
one embodiment of the invention, the enhanced image of the video mosaic is 
provided by computer system 100 in response to processor 104 executing 
sequences of instructions contained in main memory 106. Such instructions may 
be read into main memory 106 from another computer-readable medium, such as 

20 storage device 110. However, the computer-readable medium is not limited to 
devices such as storage device 110. For example, the computer-readable medium 
may include a floppy disk, a flexible disk, hard disk, magnetic tape, or any other 
magnetic medium, a CD-ROM, any other optical medium, punch cards, paper 
tape, any other physical medium with patterns of holes, a RAM, a PROM, an 

25 EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave 
embodied in an electrical, electromagnetic, infrared, or optical signal, or any other 
medium from which a computer can read. Execution of the sequences of 
instructions contained in the main memory 106 causes the processor 104 to 
perform the process steps described below. In alternative embodiments, hard- 
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wired circuitry may be used in place of or in combination with computer software 
instructions to implement the invention. Thus, embodiments of the invention are 
not limited to any specific combination of hardware circuitry and software. 

Computer system 100 also includes a communication interface 118 
5 coupled to the bus 102. Communication interface 108 provides a two-way data 
communication as is known. For example, communication interface 1 1 8 may be 
an integrated services digital network (ISDN) card or a modem to provide a data 
communication connection to a corresponding type of telephone line. As another 
example, communication interface 118 may be a local area network (LAN) card 

10 to provide a data communication connection to a compatible LAN. In the 
preferred embodiment communication interface 118 is coupled to a virtual 
blackboard. Wireless links may also be implemented. In any such 
implementation, communication interface 118 sends and receives electrical, 
electromagnetic or optical signals which carry digital data streams representing 

15 various types of information. Of particular note, the communications through 
interface 118 may permit transmission or receipt of the enhanced image of the 
video mosaic. For example, two or more computer systems 100 may be 
networked together in a conventional manner with each using the communication 
interface 118. 

20 Network link 120 typically provides data communication through one or 

more networks to other data devices. For example, network link 120 may provide 
a connection through local network 122 to a host computer 124 or to data 
equipment operated by an Internet Service Provider (ISP) 126. ISP 126 in turn 
provides data communication services through the world wide packet data 

25 communication services through the world wide packet data communication 
network now commonly referred to as the "Internet" 128. Local network 122 and 
Internet 128 both use electrical, electromagnetic or optical signals which carry 
digital data streams. The signals through the various networks and the signals on 
network link 120 and through communication interface 118, which carry the 
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digital data to and from computer system 100, are exemplary forms of carrier 
waves transporting the information. 

Computer system 100 can send messages and receive data, including 
program code, through the network(s), network link 120 and communication 
5 interface 118. In the Internet example, a server 130 might transmit a requested 
code for an application program through Internet 128, ISP 126, local network 122 
and communication interface 118. In accordance with the invention, one such 
downloaded application provides for information discovery and visualization as 
described herein. 

10 The received code may be executed by processor 104 as it is received, 

and/or stored in storage device 110, or other non- volatile storage for later 
execution. In this manner, computer system 100 may obtain application code in 
the form of a carrier wave. 

As depicted in Figures 2A and 2B there are a plurality of structures 205, 

15 210, 220, 225, 230, 240, 242, 244, 246 and 248. As depicted in photos 200 and 
250, a building structure 205 is located at the lower left hand corner of both 
frames. As is evident from frames 200 and 250, building structure 205 is not in 
the same location in each of the consecutive video frames 200, 250 because of the 
motion of the camera. Structures 215, 220 and 225 are building structures which 

20 are in the upper right hand portion of video frames 200 and 250. Structure 230 is 
a building structure having multiple sides and having a v-shaped portion 232. A 
plurality of automobiles 240-248 are parked adjacent to building structure 230. 

Referring now to Figures 3A and 3B, the various structures described in 
Figures 2A and 2B have been detected using an edge detecting process by 

25 detecting the change in intensity from one pixel to adjacent pixels. This will be 
described in greater detail below. 

Figures 4A and 4B depict consecutive frames with structures having been 
identified. Figures 5A and 5B depict consecutive frames showing matched 
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structures according to the present invention. Figures 2A-5B will now be 
explained with reference to the process of the present invention. 

Refer now to Figures 6A and 6B showing the process of the present 
invention. At step 605 the process is started. A step 610, individual frames are 

5 extracted from the library. As depicted in Figures 2A and 2B, frames 200 and 
250 were extracted from the library. At step 615, the extracted individual frames 
are converted to a black and white format. At step 620, edge detection is 
performed by detecting change and intensity from one pixel to adjacent pixels. 
As depicted in Figures 3 A and 3B, the outlines of various structures are detected. 

10 At step 625, lines are drawn at changes of intensity as depicted in Figure 3. At 
step 630, regions of interest are determined. At step 635, the regions of interest 
are correlated as explained with reference to Figures 7A and 7B. At step 640, 
image registration is performed by compensating for platform/camera movement. 
The registration process accounts for motion of the camera by determining the 

15 frame-to-frame x-y offsets, zoom and rotation. At step 645, frame overlay is 
performed. At step 650, the video mosaic can be viewed. At step 655 the process 
is ended. 

Refer now to Figure 7 were at step 705 the process is started. At step 710, 
the centroid region of interest (ROI) is calculated. At step 715, the centroid is 

20 compared with centroids of next to adjacent frame. At step 720, centroids are 
selected which are within ariel tolerances. At step 725, there is a full correlation 
of average distance from every pixel and corresponding structure. At step 730, if 
the difference is consistent the structure is identified as a potential match. At step 
735, steps 705-730 are repeated for other structures that fall within error 

25 tolerance. At step 740, the stored difference calculations are analyzed and select 
matches are based on pixels within structure having the most consistent 
differences. The analysis includes looking for frame-to-frame location as 
indicated by the difference calculation. This consistency will yield x-y 
translation, rotation and focal length changes. 
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Now referring to Figure 8, the process is started at step 805. At step 810, 
the frame is searched for an edge. At step 815, adjacent "on" pixels are followed 
until an "off pixel is detected. At step 820, it is determined the locations of the 
on pixels and these locations are stored. At step 825, the number of on pixels is 

5 counted within the structure which must exceed a preset threshold. At step 830, 
the value of the pixels within a designated structure is changed to avoid use in 
future structures. At step 835, steps 805-830 are repeated until the entire images 
in structure detected. At step 840, the process is ended. 

As depicted in Figure 9, five video frames are extracted which were taken 

10 at 30 frames a second. Thus, there are video frames 910, 920, 930, 940 and 950 
which are extracted from a data library. The date library can be onboard the 
UAV. 

As depicted in Figure 10, each of the five images is upsampled by a factor 
of 4. Referring to Figure 11, there are four pixels, 1-1, 1-5, 5-1 and 5-5 which 

15 were previously adjacent to each other but have now been upsampled by a factor 
of 4. Thus, a 5 x 5 matrix has been created. For example, pixel 1-1 has an 
intensity of .80; pixel 1-5 has an intensity of .60; pixel 5-1 has an intensity of .80; 
and pixel 5-5 has an intensity of .90. 

The images are then aligned as discussed above with respect to Figures 2- 

20 9. After the images have been aligned, then an average value from each of the 
five images in sequence is taken as depicted in Figure 12. In Figure 12, there are 
a series of five images 1210, 1220, 1230, 1240 and 1250 which correspond to the 
images 910, 920, 930, 940, 950. These five images are then combined and 
averaged into a single image as depicted in Figure 13. 

25 Referring now to Figure 14, a flow diagram summarizing the steps of the 

present invention is illustrated. At step 1405, the process is started. At step 1410, 
a low resolution image registration is performed for a sequence of images. At 
step 1415, each of the images upsampled. As depicted in Figures 10 and 11, the 
images are upsampled by factor 4. Other upsampling factors could be used, but 
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the factor of 4 appears to be optimal, yielding the most consistent results. Less 
than 4 could be used, but the idea is to create the highest resolution possible. 
Using a factor greater than 4 will result in diminishing returns, in that the amount 
of memory and processor capacity required will not necessarily product a 

5 sufficiently higher quality image. At step 1420, an x, y registration is performed 
for the upsampled images. At step 1425, the upsampled images are then aligned. 
Using a simple correlation technique to determine the x-y frame-to-frame offsets. 
At step 1430, these aligned, upsampled images are then combined into a high 
resolution output image by performing a pixel-by-pixel average across all 5 of the 

10 upsampled aligned images. At step 1435, the process is ended. 

It will be readily seen by one of ordinary skill in the art that the present 
invention fulfills all of the objects set forth above. After reading the foregoing 
specification, one of ordinary skill will be able to affect various changes, 
substitutions of equivalents and various other aspects of the invention as broadly 

15 disclosed herein. It is therefore intended that the protection granted hereon be 
limited only by the definition contained in the appended claims and equivalents 
thereof. 
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What is claimed is: 

1. A computer-implemented method of enhancing a video image, 
comprising; 

extracting a sequence of video frames; 
upsampling each of the video frames; 
interpolating the upsampled video frames; 
aligning the interpolated video frames; and 
creating a single image from the aligned video frames. 

2. The method of claim 1, wherein the sequence of video frames are 
low resolution images. 

3. The method of claim 1, wherein said upsample step is by a factor 

of 4. 

4. The method of claim 1, wherein said align step comprises aligning 
the video images in an x direction and a y direction in a center portion of interest 
in each video frame. 

5. The method of claim 1, comprising extracting the sequence of 
video frames at 30 frames/sec. 

6. The method of claim 1, wherein the sequence of video frames 
includes 5 video frames. 

7. The method of claim 1 ? comprising correlating the upsampled 
video images. 
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8. The method of claim 7, comprising averaging a pixel intensity 
from each of the upsampled video frames. 

9. The method of claim 1, comprising compensating for platform 
movement and rotation zoom. 

10. The method of claim 1, comprising aligning each the extracted 
sequence of video frames before said upsample step. 

11. The method of claim 1, comprising identifying commonality from 
one individual frame to the next and overlapping the individual frames and 
displaying an image representing a continuous area. 

12. The method of claim 11, comprising extracting the sequence of 
video frames at 30 frames/sec. 

13. The method of claim 12, wherein the sequence of video frames 
includes 5 video frames. 

14. The method of claim 13, comprising correlating the upsampled 
video images. 

15. The method of claim 14, comprising averaging a pixel intensity 
from each of the upsampled video frames. 

16. The method of claim 15, comprising compensating for platform 
movement and rotation zoom. 
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17. The method of claim 16, comprising aligning each the extracted 
sequence of video frames before said upsample step. 

18. The method of claim 17, comprising identifying commonality from 
one individual frame to the next and overlapping the individual frames and 
displaying an image representing a continuous area. 

19. The method of claim 18, comprising extracting the sequence of 
video frames at 30 frames/sec. 

20. A computer architecture, comprising: 
extracting means for extracting a sequence of video frames; 
upsampling means for upsampling each of the video frames; 
interpolating means for interpolating the upsampled video frames; 
aligning means for aligning the interpolated video frames; and 
creating means for creating a single image from the aligned video frames. 

21. An article, comprising: 

at least one sequence of machine executable instructions; 

a medium bearing the executable instructions in machine form, wherein 
execution of the instructions by one or more processors causes the one or more 
processors to: 

extract a sequence of video frames; 

upsample each of the video frames; 

interpolate the upsampled video frames; 

align the interpolated video frames; and 

create a single image from the aligned video frames. 
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22. A computer system, comprising: 
a processor; and 

a memory coupled to said processor, the memory having stored therein 
sequences of instructions, which, when executed by said processor, causes said 
5 processor to perform the steps of: 

extract a sequence of video frames; 
upsample each of the video frames; 
interpolate the upsampled video frames; 
align the interpolated video frames; and 
10 create a single image from the aligned video frames. 
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ABSTRACT OF THE DISCLOSURE 



Disclosed is a computer-implemented method of enhancing a video image. 
A sequence of video frames is extracted. Each of the video frames is upsampled. 
The upsampled video frames are interpolated. The interpolated video frames are 
aligned and a single image is created from the aligned video frames. 
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J Hauptman Reg. No. 29,3 10; Kenneth M. Berner, Reg. No. 37,093, Michael G. Oilman, Reg. No. 19,1 14; and Randy A. Noranbrock, 
Reg No 42 940 all of LOWE HAUPTMAN GOPSTEIN OILMAN & BERNER, LLP, 1700 Diagonal Road, Suite 310, Alexandria, 
Virginia'22314 'and Gerald L. Lett, Reg. No. 24,509 and Keith M. Wilson, Reg. No. 40,230, with full power of substitution and 
revocation to prosecute this application and to transact all business in the Patent and Trademark Office connected therewith, and al future 
correspondenceshouldbe addressed to Lowe Hauptman Gopstein Oilman & Berner, LLP., c/o Kenneth M. Berner, 1700 Diagonal Road, 
Suite 310, Alexandria, VA 22314. 

********************************************************* ********************************************** 

Full name of sole or first i nventor: Thomas S. Heath 

Inventor's signature: 

Residence: Syrac use, NY — 

Citizenship: United States ^ 



Post Office Address: 412 Plymouth Drive, Syracuse, NY 13206 



